A comprehensive study of different image super resolution reconstruction algorithms

INTERNATIONAL JOURNAL OF ADVANCED RESEARCH IN
International Journal of Advanced Research in Engineering and Technology (IJARET), ISSN 0976 –
6480(Print), ISSN 0976 – 6499(Online) Volume 4, Issue 1, January - February (2013), © IAEME
ENGINEERING AND TECHNOLOGY (IJARET)
ISSN 0976 - 6480 (Print)
ISSN 0976 - 6499 (Online) IJARET
Volume 4, Issue 1, January- February (2013), pp. 124-133
© IAEME: www.iaeme.com/ijaret.asp ©IAEME
Journal Impact Factor (2012): 2.7078 (Calculated by GISI)
www.jifactor.com

A COMPREHENSIVE STUDY OF DIFFERENT IMAGE SUPER-
RESOLUTION RECONSTRUCTION ALGORITHMS

Netra Lokhande Dr. Dinkar M Yadav
Research Scholar (JJT UNIV) Research Guide ( JJT UNIV)
Assistant Professor, Electrical Principal,
KJCOEMR, Pune (India) B.S.I.O.T.R(W),Pune(India)

ABSTRACT

Super-resolution image reconstruction produces a high-resolution image from a set of
shifted, blurred, and decimated versions thereof. Super-resolution image restoration has
become an active research issue in the field of image restoration. In general, super-resolution
image restoration is an ill-posed problem. Prior knowledge about the image can be combined
to make the problem well-posed, which contributes to some regularization methods. In these
regularization methods, however, regularization parameter was selected by experience in
some cases. Other techniques to compute the parameter had too heavy computation cost. This
paper presents a generalization of restoration theory for the problem of Super-Resolution
Reconstruction (SRR) of an image. In the SRR problem, a set of low quality images is given,
and a single improved quality image which fuses their information is required. We present a
model for this problem, and show how the classic restoration theory tools-ML, MAP and
POCS-can be applied as a solution. A hybrid algorithm which joins the POCS and the ML
benefits is suggested.

Key Words: Image Reconstruction, Super-Resolution, Algorithms, Regularization, Inverse
Problem, POCS, MAP Image Reconstruction.

1. INTRODUCTION

The Classic theory of restoration of a single image from linear blur and additive noise
has drawn a lot of research attention in the last three decades [1]–[4]. In the classic
restoration problem in image processing, a blurred and noisy image is given and the purpose

124


is to somehow restore the ideal image prior to the degradation effects. Such problem is
typically modeled using the linear vector-matrix equation

Y = H X + N ----------------------- (1)

Many algorithms were proposed in the literature for this classic and related problems,
contributing to the construction of a unified theory that ties together many of the existing
methods [4]. In the single image restoration theory, three major and distinct approaches are
extensively used in order to get practical restoration algorithms:
1) Maximum likelihood (ML) estimator [1]–[4],
2) Maximum a posteriori (MAP) probability estimator [1]–[4], and
3) Projection onto convex sets (POCS) approach. Super-resolution techniques offer a
possibility to produce an image with a higher resolution from a set of images with lower
resolution. The underlying mechanism is implied in the fact that different sub-pixel
displacement of each low resolution images contains different information of the high
resolution image. The pioneer work of super-resolution reconstruction may go back to 1984
by Tsai and Huang. Since then, many researchers have devoted themselves to the work in this
area. Currently, the research is focused to a few points, such as high precision sub-resolution
registration algorithm, blind super-resolution methods, robust and efficient reconstruction,
real-time processing techniques.
The goal of Super-resolution restoration is to reconstruct the original scene from a
degraded observation. By "Super-resolution", we refer to removal of blur caused by the
image system (out of focus blur, motion blur, non-ideal sampling, etc.) as well as recovery of
spatial frequency information beyond the diffraction limit of the optical system. This
recovery process is critical to many image processing applications. And extracting a high
resolution image from some low resolution image is required in many facets of image
processing. For example, in remote sensing field, where several images of the same area are
given, and an improved resolution image is required; or in video processing, where single
frame in video signal is generally of poor quality. Enhancement of a single image can be
done by using several successive images merged together by a super-resolution algorithm.
Super-resolution restoration from a still image is a well recognized example of an ill posed
inverse problem. Such problems may be approached using regularization methods that
constrain the feasible solution space by employing a-priori knowledge. This may be achieved
in two complimentary ways.(1) obtain additional novel observation data and (2) constrain the
feasible solution space with a-priori assumptions on the form of the solution. We identify
three critical factors affecting super-resolution restoration. Firstly, reliable sub-pixel motion
information is essential. Secondly, observation models must accurately describe the imaging
system and its degradations. Thirdly, restoration methods must provide the maximum
potential for inclusion of a-priori information. In tradition single image restoration problem
only a single input image is available for processing. Super-resolution image restoration
addresses the problem of producing super-resolution still image from several images, which
contains additional similar, but not identical information. The additional information makes it
possible that construct a higher resolution image form original data. Super-resolution
techniques can be divided into two main divisions: frequency domains and spatial domain.
Frequency domain methods are earlier super-resolution methods, they can only deal with
image sequences with global translational. Spatial domain methods are very flexible. At

125


present, they are main research direction of super-resolution. Spatial methods include Iterated
Back projection (IBP), Projection onto Convex Sets (POCS), Maximum Posteriori (MAP)
estimation and Maximum Likelihood (ML) estimation. Two powerful classes of spatial
domain methods are POCS and MAP.
The three approaches merge into one family of algorithms, which generalizes the
single image restoration theory [1]–[4] on one hand, and the existing super-resolution
algorithms proposed in the literature [5]–[14] on the other hand. The proposed restoration
approach is general but assumes explicit knowledge of the linear space- and time-variant blur,
the (additive Gaussian) noise, the different measured resolute ions, and the (smooth) motion
flow. The presented methodology also enables the incorporation of POCS into the ML or
MAP restoration algorithms, similar to the way it is done for the iterative single image
restoration problem [4], yielding hybrid super-resolution restoration algorithm with further
improved performance and assured convergence.

2. MODELLING SUPER- RESOLUTION PROBLEM

Given are N measured images [Y k]k =1to N of different sizes [Mk ×Mk ]. We assume
that these images are different representations of a single high-resolution image X of size [L
× L], where typically, L > Mk for 1≤ k ≤ N. Each measure image is the result of an arbitrary
geometric warping [L2 × L2 ] matrix Fk, linear space variant blurring [L2 × L2 ] matrix Ck and
uniform rational decimating [Mk2 x L2 ] matrix Dk performed on the ideal high-resolution
image X. We further assume that each of the measured images is contaminated by zero mean
additive Gaussian noise vector E k with auto-correlation [Mk2 x Mk2] matrix Wk−1.These noise
vectors are uncorrelated between different measurements. Translating the above description
to an analytical model as in Fig.1. we get,

Y k = Dk Ck Fk X + E k for 1≤ k ≤ N --------- (2)

Fig.1. Degradation Model for the Super-Resolution Problem

126


All these matrices (Fk , Ck , Dk , Wk ) are assumed to be known in advance. Justifying such an
assumption is treated in [15]. Having the above model, grouping the N equations into one can
be done for notational convenience. This way we get:

Y=HX+E ---------------------- (3)

Where we have defined Hk= Dk Ck Fk, and the autocorrelation of the Gaussian random vector
E is

---------------------- (4)

The obtained model equation Y = H X + E is a classic restoration problem model [1]–[4].
Thus, we can easily apply the Maximum Likelihood (ML) estimator, the Maximum A-
posteriori (MAP) estimator or the Projection Onto Convex Sets (POCS) methods in order to
restore the image X , which is exactly our purpose here. In the following sections we shall
briefly present the way to apply each of those tools.

3. ML RESTORATION

Applying the ML solution [15] we get

------------------------------- (5)

This gives the well known pseudo- inverse result

------------------------------ (6)
Where

----------------------------- (7)

127


Locally adaptive regularization can be included in the above analysis with both algebraic and
physical interpretations. Using the Laplacian operator S and a weighting matrix V (penalizing
non smoothness according to the a priori knowledge on the smoothness required at each
pixel), we get

----------------(8)

Differentiating again with respect to X and equating to zero yields the
equation, which is the same as in (6), but a new term β STV S, , is added to the
matrix R.

4. MAP RESTORATION

If we assume that the unknown X is a zero mean Gaussian random process with auto-
correlation-matrix Q, the MAP estimator becomes the Minimum Mean Square Error
estimator. Performing several algebraic steps [6] gives,

--------------------- (9)

Minimizing the above function with respect to yields the following result

--------------------- (10)

Where

-------------- (11)

And the resemblance to the ML result is evident. It can be shown [4] that if an autoregressive
(AR) model is assumed on the image, a simple and direct connection between the Laplacian
regularization matrix and the AR coefficients can be established.
The ML, the MAP estimator reduces to a huge sparse set of equations which can be solved
iteratively.

128


5. SET THEORETIC RESTORATION

According to the set theoretic approach [5], each a- priori knowledge on the required
restored image should be formulated as a constraining convex set containing the restored
image as a point within this set. Using the model presented earlier, we can suggest a group of
such convex sets based on L2 distance measure

------------------------ (12)

for 1≤ k ≤ N. This defines a group of N convex sets - ellipsoids in this case. Since POCS
requires a projection onto these sets, and since projection onto an ellipsoid is computationally
very complex, L∞ constraints can be proposed instead [15].

--------------- (13)

where θk is the support region of the k-th measured image, and δk stands for the uncertainty of
the model [16], [17]. Another set which can be used is the one constraining smoothness. We
can suggest L2 or L∞ convex set versions as before:

--------------------- (14)

or

------------------------ (15)

where θ0 is the support region of the ideal image. We can incorporate additional nonlinear
constraints such as constraints on the output energy, phase, support, and others. An often used
constraint is the one posed on the amplitude of the result

---------------------------- (16)

Having a group of M convex sets, each containing the required image, the Projection Onto
Convex Set (POCS) method suggests the following iterative algorithm for the recovery of a
point within the intersection of these sets [1,15]

-------------------------- (17)

where Pj is the projection of a given point onto the jthconvex set.

129


A different approach towards the POCS idea is the bounding ellipsoid method [15].For the
case where all the constraints are ellipsoids this approach suggests finding the ellipsoid
bounding the intersection of all the participating constraints, and to choosing its center as the
output result. In [15] it is shown that the equation for the bounding ellipsoid center is exactly
(for a specific case) the ML solution as given in equation (8).

6. HYBRID RESTORATION

While the ML and the MAP are numerically simpler to apply, the POCS is more
general and can incorporate non-linear constraints into their construction process as well. In
order to gain both these properties, a hybrid algorithm is proposed. We start by defining a
new convex optimization problem which combines a quadratic scalar error with M convex
constraints.

---------------------------- (18)

subject to { X ε Hk 1≤ k ≤ M }
where the quadratic error takes care of the model and the smoothness errors, and the M
additional constraints refer to the non-ellipsoids a-priori knowledge.
Following the iterative methods presented by [1],we propose a simple yet effective two-phase
iterative algorithm to solve the above optimization problem. Analysis of this method can be
found in[15]. Suppose that an efficient iterative algorithm which is known to converge to the
minimum of the scalar squared error is given - denoted by It. Algorithms such as the
Conjugate Gradient or the Gauss-Siedel can be considered as excellent candidates for It.
Beyond this first iterative algorithm It, M projection operators denoted by Jtk k= 1,2. . . M can
be constructed; each of them projects onto a convex set representing a given constraint.
Assuming that the M projections are all given using the Euclidean metric, we suggest the
following global iterative step,

------------------------------- (19)

This interlaced approach is generally converging to the sub-optimal point of the problem
given in equation (18).Adding several new iterations, where now It is replaced by the
[19]Steepest Descent, updates the previous result and assures that the final convergence is to
the optimal point, as is proved in [18].

7. SIMULATION RESULTS

In this section we present elementary example which demonstrate the effectiveness of
the proposed method for the super-resolution restoration problem. A single 100 ×100 image
was taken (gray values in the range 0-63), and from it we have generated 16 blurred, down-
sampled and noisy images of size 50 × 50. The degradation includes random affine motion
(with zoom in the range (0.9.1.1), rotation in the range (0,50°), and translation in the range -
5,5) pixels, blur with the 1-D separable kernel (0.7 1.0 0.7) / 2.4, a 2:1 decimation ratio, and
additive white Gaussian noise with σ = 3.

130


Fig.1 presents the ideal image, Fig.2 presents 4 images from the measurements, and Fig.3
shows the reconstructed image using the hybrid restoration algorithm with regularization.

Fig.1 - The ideal image

Fig.2 - Four images from the measurements

Fig.3 - A reconstructed result

131


8. CONCLUSION

This paper addresses the Super-Resolution Reconstruction problem and its solution,
where given a number of moved, blurred, and noisy versions of a single ideal image, and one
wants to restore the original image. To solve this problem, a new general model was
introduced here. This model enabled the direct generalization of classic tools from restoration
theory to the new problem. In this context, the ML, the MAP, and the POCS methods are all
shown to be directly and simply applicable to super-resolution restoration with equivalencies
between these methods. The restoration problem at hand in each of these approaches reduces
to the problem of solving a very large set of sparse linear equations. A hybrid algorithm is
proposed that combines the benefits of the simple ML estimator, and the ability of the POCS
to incorporate non ellipsoids constraints. This hybrid algorithm solves a constrained convex
minimization problem, combining all the a priori knowledge on the required result into the
restoration process. An efficient iterative two-phase algorithm is presented for solving the
defined problem, and convergence is assured to the optimal point. Simulations are performed
to demonstrate super-resolution restoration using the hybrid algorithm.

9. REFERENCES

[1].R.C.Gonzalez and P.Wintz, Digital Image Processing. New York: Addison-Wesley, 1987.
[2] W. K. Pratt, Digital Image Processing. New York: Wiley, 1991.
[2]. W. K. Pratt, Digital Image Processing. New York: Wiley, 1991.
[3]. A. K.Jain, Fundamentals in Digital Image Processing. Englewood Cliffs, NJ: Prentice-
Hall, 1989.
[4]. R.L.Lagendijk and J. Biemond Iterative Identification and Restoration of Images. Boston,
MA: Kluwer, 1991.
[5]. T. S. Huang and R. Y. Tsay, “Multiple frame image restoration and registration,” in
Advances in Computer Vision and Image Processing, vol. 1, T. S. Huang, Ed. Greenwich,
CT: JAI, pp. 317–339,1984.
[6]. S. P. Kim, N. K. Bose, and H. M Valenzuela, “Recursive reconstruction of high
resolution image from noisy under sampled multi-frames,” IEEE Trans. Acoust., Speech,
Signal Processing, vol. 38, pp. 1013–1027, June,1990.
[7]. N.K.Bose, S. P. Kim, and H. M Valenzuela, “Recursive implementation of total least
squares algorithm for image reconstruction from noisy, under sampled multi-frames,” in
Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), Minneapolis
MN, vol. V, pp. 269–272,1993.
[8]. H. C. Kim, “High resolution image reconstruction from under sampled multi-frames,”
Ph.D. dissertation, Pennsylvania State Univ., Univ. Park,PA, 1994.
[9]. A. Papoulis, “Generalized sampling theorem,” IEEE Trans. Circuits Syst., vol. CAS-24,
pp. 652–654, Nov. 1977.

[10]. L. J. Yen, “On non uniform sampling of bandwidth limited signals,” IRE Trans. Circuits
Theory, vol. 3, pp. 251–257, Apr. 1956.
[11]. H. Ur and D. Gross, “Improved resolution from sub-pixel shifted pictures,” CVGIP:
Graph. Models Image Process., vol. 54, pp. 181–186,Mar. 1992.
[12]. C. Srinivas and M. D. Srinath, “A stochastic model-based approach for simultaneous
restoration of multiple mis-registered images,” SPIE, vol.1360, pp. 1416-1427, 1990.
132


[13]. S. Peleg, D. Keren, and L. Schweitzer, “Improving image resolution using subpixel
motion,” Pattern Recognit. Lett., vol. 5, pp. 223–226,Mar. 1987.
[14]. M. Irani and S. Peleg, “Improving resolution by Image Registration,”CVGIP: Graph.
Models Image Process., vol. 53, pp. 231–239, Mar.1991.
[15]. A. K. Katsaggelos, “A multiple input image restoration approach,” J.Vis. Commun.
Image Representat., vol. 1, pp. 93–103, Sept. 1990.
[16]. A. M. Tekalp, M. K. Ozkan, and M. I. Sezan, “High-resolution image reconstruction
from lower-resolution image sequences and space varying image restoration,” IEEE Int.
Conf. Acoustics, Speech and Signal Processing (ICASSP), San Francisco, CA., Mar.
1992, vol. III, pp.169–172.
[17] A. J. Patti, M. I. Sezan, and A. M. Tekalp, “High-resolution image reconstruction from a
low-resolution image sequence in the presence of time-varying motion blur,” in Proc.
ICIP, Austin, TX, Nov. 1994, pp.343-347.
[18]. M. Elad and A. Feuer, "Restoration of Single Super-Resolution Image From Several
Blurred Noisy and Under-Sampled Measured Images", Submitted to the IEEE Trans.
Image Processing on July 1995.
[19].L.A.Hageman and D.M.Young, Applied iterative methods, Academic Press, New-York,
1981.
[20] R. Pushpavalli and G.Sivaradje, “Nonlinear Filtering Technique For Preserving Edges
and Fine Details Of Digital Images” International journal of Electronics and
Communication Engineering &Technology (IJECET), Volume 3, Issue 1, 2012,
pp. 29 - 40, Published by IAEME.
[21] Reena M Patel and D J Shah, “Concealogram: Digital Image in Image Using LSB
Insertion Method” International journal of Electronics and Communication Engineering
& Technology (IJECET), Volume 4, Issue 1, 2013, pp. 230 - 235, Published by IAEME.
[22] Ankit Vidyarthi and Ankita Kansal, “A Survey Report On Digital Images Segmentation
Algorithms” International journal of Computer Engineering & Technology (IJCET),
Volume 3, Issue 2, 2012, pp. 85 - 91, Published by IAEME.
[23] Mr.J.Rajarajan and Dr.G.Kalivarathan, “Influence of Local Segmentation in the Context
of Digital Image Processing – A Feasibility Study” International journal of Computer
Engineering & Technology (IJCET), Volume 3, Issue 3, 2012, pp. 340 - 347, Published
by IAEME.

133

A comprehensive study of different image super resolution reconstruction algorithms

More Related Content

What's hot

Viewers also liked

Similar to A comprehensive study of different image super resolution reconstruction algorithms

More from IAEME Publication

A comprehensive study of different image super resolution reconstruction algorithms